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Solving the error correcting code is an important goal with regard to communication theory. To reveal the 
error correcting code characteristics, several researchers have applied a statistical-mechanical approach to this 
problem. In our research, we have treated the error correcting code as a Bayes inference framework. Carrying 
out the inference in practice, we have applied the NMF (naive mean field) approximation to the MPM (maxi- 
mizer of the posterior marginals) inference, which is a kind of Bayes inference. In the field of artificial neural 
networks, this approximation is used to reduce computational cost through the substitution of stochastic binary 
units with the deterministic continuous value units. However, few reports have quantitatively described the per- 
formance of this approximation. Therefore, we have analyzed the approximation performance from a theoretical 
viewpoint, and have compared our results with the computer simulation. 



I. INTRODUCTION 

Within the framework of an error-correcting code, a sender 
encodes a message with redundant information added to 
the transmitted sequence, and a receiver obtains the noise- 
corrupted sequence through a noisy transmitting channel. De- 
coding process of the error-correcting code is to restore the 
transmitted message from the received sequence that is cor- 
rupted through the transmitting process. 

Sourlas suggested that the error-correcting code can be 
dealt with Bayesian inference, and proposed an encoding 
model where all possible combinations of r-bits must be mul- 
tiplied as the redundant information just like the Mattis model 
d. Sourlas also showed that the channel capacity of the 
model can reach the theoretical limit, called the Shannon 
bound 1 2] in the limit of r — > oo. Unfortunately, under the 
condition r — > oo, the transmitting speed becomes to 0, so 
that the model with large r is not practical. Recently, how- 
ever, Kabashima & Saad pointed out that the Sourlas encod- 
ing model with a small r, - for example r = 2, 3 — , is capable 
of good restoration ability with a practical transmitting speed 

a. 

In this paper, we also treat this problem according to a 
Bayesian inference framework. This framework is based on 
estimating the restored message occurrence probability (pos- 
terior) from both the prior probability meaning the original 
message generation probability and the likelihood, which de- 
pends on the corruption process model. 

One strategy using Bayesian inference is to use a message 
which maximizes the posterior probability. This method is 
called maximum a posteriori (posterior) probability (MAP) 
inference. Given a corrupted sequence, the MAP inference 
accepts as the restored result the message which maximizes 
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the posterior probability. From the statistical mechanical point 
of view, the logarithm of the posterior probability can be re- 
garded as the energy, so we can consider the MAP inference 
as an energy minimization problem. 

Another strategy is to use inference in which we consider 
the expectation value with respect to the maximized marginal 
posterior probability at each site's thermal equilibrium as the 
original message. This method is called maximizer of the 
posterior marginals (MPM) inference|4]|5]|6]. From the sta- 
tistical mechanical point of view, the MPM inference corre- 
sponds to minimization of the free energy. In the MAP infer- 
ence, the posterior probability is given for each candidate of 
sequences. In contrast, in the MPM inference, the posterior 
marginal probability is given for each bit in the sequence, and 
the state which has the largest posterior probability is chosen 
as the restored state for each bit. Hence, to find a optimal bit 
sequence through the MPM inference, we should compare the 
posterior probability for each bit, and this requires that we cal- 
culate the thermal average for each sequence bit. Recently, the 
MPM inference has been discussed with regard to error cor- 
recting code 1 4] 1 5] 1 6]. Rujan pointed out the effectiveness 
of carrying out a decoding procedure not at the ground state, 
but at a finite temperature|4|. Sourlas used the Bayes for- 
mula to re-derive the finite-temperature decoding of Rujan's 
result under more general conditions |5]. Finite-temperature 
decoding corresponds to the MPM inference, while decoding 
at the ground state corresponds to the MAP inference. Com- 
parison of the restoration ability of the MPM inference to that 
of the MAP inference with regard to the error rate per bit has 
shown that, the MPM inference is superior to the MAP infer- 
ence HG1. 

To consider the error correcting code from a statistical me- 
chanical perspective, we denoted the message length, where 
the message is represented by a binary unit sequence, as N, 
and assumed that each unit can take a binary state { — 1, +1}. 
The number of feasible message combinations in this case 
would reach 2 , making it hard to find a correct message 
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FIG. 1 : Schematic diagram of a framework for the error correcting code. 
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among the numerous candidates. Thus, to apply the MAP or 
MPM inference in a practical way, we usually have to adopt 
some kind of gradient descent algorithm, such as the Glauber 
dynamics, despite a risk that the solution will be captured 
by local minima. Moreover, applying the MPM inference 
may take more computational time than applying the MAP 
inference. In the MAP inference, once the system reaches a 
macroscopic equilibrium state, each pixel value can be prop- 
erly determined with a probability of 1. On the other hand, 
when the system reaches a macroscopic equilibrium state in 
the MPM inference, each unit value cannot be deterministicly 
decided because the probabilities for each binary state have 
finite values in finite-temperature decoding. Therefore, we 
need to calculate the thermal averages for each unit, and this 
requires many samplings. In this paper, we discuss an approx- 
imation that replaces the stochastic binary units with deter- 
ministic analog units which can take [— 1,+1] continuously. 
In other words, we introduce deterministic dynamics into the 
MPM inference to avoid the need to sample for the thermal 
average. In statistical mechanics, this approximation is some- 
times called the "naive mean field (NMF)" approximation 0]. 
The purpose of the NMF approximation has usually been to 
enable deterministic analog units to emulate the behavior of 
stochastic binary units. This approximation has been applied 
to several combinatorial optimization problems, such as the 
traveling salesman problem (TSP) |8]. 

One important advantage of applying the NMF approxima- 
tion is that the calculation cost is reduced since we can avoid 
calculating the thermal average which requires many sam- 
plings in stochastic dynamics. However, the NMF approx- 
imation has typically been applied as a mere approximation 
in algorithm implementation, so few researchers who have 
used the NMF approximation have investigated its quantita- 
tive ability - e.g., the approximation accuracy. In this paper, 
we discuss the quantitative ability of the NMF approximation 
in the MPM inference|9]. 

Nishimori & Wong formulated the image restoration prob- 
lem and the error correcting code from the statistical- 
mechanical perspective by introducing mean field models for 
binary image restoration, and analyzed this model theoreti- 
cally through the replica method. We have applied the NMF 
approximation to the formulation, and analyzed the model 



through the replica method in the manner of Bray et al 1 7 ] . 

In the field of neural networks, a network model using the 
NMF approximation is sometimes called an analog neural net- 
work. Roughly speaking, Hopfield & Tank's network is a 
result of applying the NMF approximation to the optimiza- 
tion network proposed by Kirkpatrick 1 10] 1 8]. Therefore, a 
Hopfield-Tank type network can be regarded as a kind of ana- 
log neural network. The formulation of this kind of optimiza- 
tion network is very similar to that of the Sourlas encoding 
model with r = 2. In our study, we applied the NMF approx- 
imation to the Sourlas encoding model with r > 2, which is 
called 'r-body interaction' from a statistical-mechanical view- 
point, so that our formulation can be considered a natural ex- 
tension of a Hopfield-Tank type network with higher-order di- 
mension interactions. 

In Sec. II, we formulate the error-correcting code us- 
ing Bayes inference in the manner of Nishimori & Wong's 
formulation|6|. 

In Sec. EH, we compare the results between from our analy- 
sis to those of a computer simulation. Within the limits of the 
mean field approximation, our results agreed with the simula- 
tion results. 



II. MODEL AND ANALYSIS 

A. Formulation of the Error Correcting Code 

In this section, to make this paper self-contained, we ex- 
plain the error correcting code in the manner of Nishimori & 
Wong |fj. Fig. ^ shows a schematic diagram of the error 
correcting code. On the sender side, the original signal is rep- 
resented by £, and each element is a binary unit that takes 
binary states £j = { — 1, +1} for i — 1 • • ■ N. The number of 
elements means the message length which can be denoted as 
N. 

Through the transmission channel, the signals being sent 
are degraded by noise, so redundant information is needed to 
enable retrieval of the original message. The error-correcting 
code has been discussed from a Bayesian inference point of 
view 1 4] 1 5] 1 6]. In the manner of the Sourlas code |5[, the 
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redundant message is produced from the r-units product: 

ex, •••;-„. (i) 

where the indices satisfy 1 < ii < i% < • • • < i r < N. 
The sender transmits the redundant message £j t £j 2 • • • £j r for 
all possible combinations of the indices. Thus, the redundant 
message length becomes ArC r . In addition, we assumed that 
the original message {^} has a uniform prior probability: 



Ps(0 = i 



for any 



(2) 



On the receiver side, degraded signals are observed since 
transmission channels ares noisy. In this paper, received sig- 
nals t correspond to the original message £ (whose elements 
consists of (£ = 1 • • ■ N)) and their elements are denoted as 
Tj (i = 1 • • • N), while signals J (whose elements consists of 
Jii---v(l — *! < ' ' ' < *f — ^V)) correspond to the redundant 
message whose elements consists of ^ 1 • • • £j r . The receiver 
should be able to estimate the original message £ from the 
received signals r, J. 

To apply Bayesian inference to estimate the original mes- 
sage, we should consider the posterior probability based on 
observation: 



P out (J,T|QP s (Q 
Tr {€} P out (J,x|OP s (0 



(3) 



where P ou t{J, t|£) is a conditional probability of the ob- 
served signal which can be regarded as the probability expres- 
sion of the corrupting process in the communication channel. 

In this study, we assumed the communication channel is a 
Gaussian channel: 



P m t(J,T\S)=P{J\t)P{T\Z) 
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(5) 
(6) 



The first term in Eq.@ corresponds to the noise of the redun- 
dant message channel, and the second term corresponds to the 
noise of the original message channel. The random variable 
Ji x ...i T is i.i.d. according to a normal distribution whose mean 
is &i ' ' ' 6i r - an d whose variance is 2 J 2 r\/N r . The 
random variable Tj is i.i.d. according to a normal distribution 
whose mean is ro£» and whose variance is r 2 . Thus, we can 
describe the transmission channel characteristics using the pa- 
rameters jo, J, to, and r. 

Substituting Eqs.© and (|4} into Eq.Q, we can denote the 



posterior probability as 

PW,t) = 
1 ( 2jo 

\ ii<---<i r 

z 



(7) 



Tr {(T} exp ( ^ ^ Jii-vfii-'^r + ^X] Ti ^ 

(8) 

where Z is a partition function. 

To distinguish the restored signal from the original one, 
which is denoted £, we use the notation <x for the restored 
units. Moreover, the noise channel characteristics, denoted by 
2jo/J 2 , and tq/t 2 in Eq..@, are not given on the receiver 
side, so the receiver should estimate these terms, which are 
called hyper-parameters. We refer to describe these hyper- 
parameters as (3, and h respectively. The posterior probability 
can thus be described as 




Z = Tr {cr } exp (3 



h nai J 

»=i / 

• cTj r + h Tj< 



(9) 



(10) 



From Eq. (|9}, it is natural to introduce a Hamiltonian de- 
scribed as 



(3H = -f3 



<!<•••<! 



(ii) 



If we ignore the original message channel -, that is h = 
-, the Hamiltonian consists of only the r-body interaction 
term; i.e., only the redundant message is effective for restora- 
tion, which is called the Sourlas code. Sourlas introduced the 
Hamiltonian and discussed the macroscopic property of the 
error correcting code using statistical analysis [5]. 

Rujan and Nishimori & Wong have applied the MPM in- 
ference to the error-correcting code as the restoration strategy 
|6]|4], that is adopting the 0$ which maximizes the marginal 
probability: 



P(ai\J,r) = Tr {(T . } . ? 



exp(-/3 J ff) 



^Tr {(T . } exp(-/?iO 
In this case, this is equivalent to adopting <ii as 

<Ti = Sgn (P((Ti = +l\J,T)-P(<Ti = -1\J,T)) 

= sgn (Tr ai aiP(ai\J,T)) 

/Tr { } o"i exp(-/3if) ^ 

= sen I — 

V Tr Wj} exp(- (3H) 

= sgn (<ri)0 



(12) 



(13) 
(14) 



The term ((Ji)p in Eq. dl4> represents the thermal average of 
<ji in the Hamiltonian H(in t\l 1|) ) with the finite temperature 
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(3. Therefore the MPM inference is called finite-temperature 
decoding. 

Nishimori and Wong compared the MPM and MAP infer- 
ences which used the cr that minimized the Hamiltonian H 
in Eg. (II 1> [6]. They pointed out that the MAP inference is 
equivalent to the MPM inference within the limit of the tem- 
perature T(= ft- 1 ) — > 0. They also suggested that the MPM 
inference is superior to the MAP inference with regard to the 
overlap: 



M„ = 



(15) 



= Tr {c} , {J}iM P s (OP out (J, t|£) d sgn^. (16) 

In Eq.jl5>. the bracket [•] , which is called as the configuration 
average, denotes the averages over the sets {£}, {J}, and {x}. 
Restoration ability measured in terms of overlap is maximized 
when the parameters and h equal, respectively, 2 jo / J 2 and 
ro/r 2 0. 

In the MPM inference, each restored unit cr,, which is a 
stochastic binary unit, is subject to thermal fluctuation since 
the decoding is carried out at a finite temperature; this means 
each <Zj = ±1 state has finite probability value. Therefore, 
we should calculate the thermal averages for all of the units 
when decoding is carried out. In contrast, decoding using 
the MAP inference is done at the temperature /3 _1 = 0, so 
there is no thermal fluctuation does not occurred. When the 
MPM inference is applied to information processing, the cal- 
culation cost of the thermal averaging must be evaluated, and 
the averaging process requires a lot of calculation cost||9|]. To 
avoid this high cost, we have introduced the naive mean field 
(NMF) approximation. We previously reported that an image- 
restoration model using the MPM inference with stochastic 
binary units requires more than 50 times as much computa- 
tional time as a model using the NMF approximation |9|. 



B. Naive Mean Field Approximation 

In this study, in an attempt to find the ground state in a 
practical way, we introduced the NMF approximation |8]|9]. 
When the NMF approximation is used, each stochastic binary 
unit <Tj is replaced by an analog unit Si that can take a con- 
tinuous value [— and the output of each analog unit 
is regarded as (erj); that is, the thermal average of the corre- 
sponding binary unit output <7j can take binary states cr, = ±1 
stochastically. 

To replace the stochastic binary units with deterministic 
analog units on the receiver side, we introduced a Hamilto- 
nian for substituting Eg. (II 1> . 



0H 



-M 



(17) 

where Ij denotes an output of an analog unit which takes 
a continuous value [—1, +1] in the equilibrium state. M is 
a scaling factor described below. To analyze the model de- 
scribed by the Hamiltonian Ea. (ll7l >. we followed the manner 
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FIG. 2: Analog model for analysis: In the binary model, encoding 
unit £ i corresponds to decoding unit cr, . In contrast, the analog model 
assumes a decoding unit has M units, and the average of these M 
units is regarded as the analog unit output. 



of Bray et al. 01 . We assumed each ith site consists of M 
binary units, and the analog unit output s, can be calculated 
using the average of M binary units &i a (see Fig0. 



1 M 

= m2> 



(18) 



a=l 



For the limit M — ► oo, each output Sj can take a continuous 
value [—1, +1]. When M is a finite value, each Sj is called a 
'binominal spins' which can take — 1, — I + tjv* ■ , 1 — ^ , 1 
with a binominal distribution. We can thus introduce a 'spin 
weight function' as 



W(8t)= Tr S^MSi-^&taJ 



27TJ 



dui exp(Af(— UiSi + In 2 cosh(u,))). 



(19) 



The partition function Z can be described as 

N ,+1 

z = Ii dSiW(Si)exp(-pH) 

-/fn(g)/Tn* 

exp M(/3 Ji l -i r Si l ■ ■ ■ s ir 

\ ^ ii<---<v 

+ h TiSi — Ui§i + In 2 cosh(iti)^ j 



(20) 



At the limit M — > oo, the integrals over {ui} and {s^} in Eq. 
( I20l i can be evaluated by the saddle-point method. To derive 
the saddle-point equations, we differentiated the exponent of 
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Ea. J20t by Sj and Ui, and obtained 

AT JV 

(r - 1)! S ' 

v ; i 2 =l 

(21) 

= — Jj + tanhu,;. (22) 
To derive Ea. J21> . we assumed Ji 1 ...i r was symmetric: 

where the indices group (i[ ■ ■ ■ i' r ) is any permutation group 
of (if ■ - i r ) ■ Moreover, we assumed self-coupled term in 
./,...., equals zero: 

Jil—l — l—ir = "• 

Eliminating u, from Eas.( l21> and (I22> . we obtain 



Sj = tanh 



AT AT 



l &2 *•* V &2 °1r ~ lvl I 



i 2 = l i r = l 



For example, for r = 3, Eq.j23> can be denoted as 



§i = tanh I 7^ XI XI Ji J k + hl ~ l 



N N 



(23) 



(24) 



J=l fc=l 



where we assumed J ljk = J klj = J jkl = J kji = J ikj = 
Jjikj and tTjjj ^"ij ^7,72 

We can derive naturally a discrete synchronous updating 
rule: 



J+ 1 



tanh 







( r -l)\ ^ U " 2 '-- l r°l2 



(25) 

where s\ denotes the analog unit output at time t. Eq.( l25t does 
not include any stochastic calculation, so we refer to Eq.j25> 
as the deterministic dynamics. When the model described by 
Eq. i25i reached to the equilibrium state s°°, all units should 
satisfy Eq. i23\ . Therefore, to investigate the equilibrium state 
of dynamics (Eq. j25», we should use the analog Hamiltonian 
described by Eq.lll7t. From the idea of the NMF approxima- 
tion, each analog unit state expressed by s°° in the equilibrium 
state will correspond to (aj), i.e. sf° can be regarded as the 
thermal average of <7j. From Eq. J25t . this model follows 
the deterministic dynamics, so there is no need to calculate 
the thermal average of a stochastic unit; the expected calcula- 
tion cost is thus lower than that for the stochastic binary units 
model. 



a standard statistical-mechanical analysis tool. The MPM in- 
ference corresponds to the minimization of the free energy 
denoted as T[mZ]. Here, Z is the partition function: 



Z = Ti >ia} exp(-/?H), 



(26) 



where TL is described as Eq.( ll7> . However, it is impossible 
to evaluate [In Z] in practical, we use replica trick [In Z] = 
lim n _»o ( [Z n ] — l)/n. The n replicated partition function [Z n ] 
can be expressed as: 



\Z n ] = Tr 



n 



\ 1—1 v 

x P s (te})Pout({J il ...i r }, {nm*})Z n , 



zdr. 



(27) 



Z " = II ex P ( X J *i-*r X K*i ■ ■ ■ Kt 



+ /f X Ti X ^ k ) ' 



(28) 



where operator Tr in the last equation represents the sum over 
all states about {a" a } and {£,}. We analyzed this replicated 
partition function through the standard replica method. The 
replica symmetry solution can be described as 



11/ - 1 Dz G(z), 
q = j DzG(z) 2 , 

X : 



1 



2 72^„r-l 



(3 2 J 2 rq 



h 2 r 2 



1/2 



where 



G (,)=tanh((^" 1 



Dz zG(z), 



1/2 



(29) 
(30) 
(31) 



f} 2 J 2 



r(r - l) X q r - 2 G(z) + (3j rm r - x + hr (32) 



Using these solutions, we could obtain the overlap M G : 

N 

X 



1 N 

T7 XI ^ S S n (^) 



i=l 



Dz sgn(G(z)) 



(33) 



(34) 



If we Assume x = 0, this analysis agrees with the result ob- 
tained using stochastic binary units reported by Nishimori& 
Wong 1 6]. Thus, in our analysis, the difference in the result 
that arises from the reaction term 
Eq.(|33. 



^r(r - l) X q r - 2 G(z) in 



C. Equilibrium State Analysis 

To investigate the macroscopic property, we analyzed the 
NMF approximated model that includes the Hamiltonian de- 
scribed by Eq. dl7l through the "replica method", which is 



III. RESULT 

In this section, we compare the theoretical and simulation 
results for the conventional stochastic binary model and the 
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FIG. 3: Phase diagram of binary and analog models: The horizontal axis shows S/N ratio jo/ J, and the vertical axis shows the decoding 
temperature T/J. (a) shows the analysis result for the binary model, and (b) shows the result for the analog model. Region 'F* shows the 
retrievable area, (c) shows the difference between (a) and (b). Region 'A' is where the analog model is superior to the binary model and the 
region 'B' indicates the opposite. 



NMF approximated (deterministic analog) model. We refer 
to the decoding using the MPM inference with conventional 
stochastic binary units as the 'binary model' and to the NMF 
approximated model as the 'analog model'. In subsection 
llll Al we show the analytical results for each model. In sub- 
section IIII Bl we compare the results from these theoretical 
analyses with those from computer simulations. To compare 
these results, we configured an environment described by sev- 
eral parameters having the same value; that is, r = 3, h = 0, 
and J = 1. The condition r = 3 means a redundant message 
was generated by 3 -body £i£j£k where l<i<j<k<N. 
We use h = 0, which means the decoding was carried out us- 
ing only information J corresponding to the redundant mes- 
sage. Thus, t had no effect in our decoding models. Under 
this condition, the original message was generated by uniform 
prior probability Eq.0. The redundant messages were gen- 
erated by Eq.Q, and the message corruption process is de- 
scribed by Eq.@. 

In the simulation, to decode the original message £ from 
the corrupted signals J using the stochastic binary model, we 
used a kind of gradient descent algorithm -, that is Glauber 
dynamics, - to find the minimum state of the Hamiltonian H 
described in Eg. (II 1>. For decoding using the analog model, 
we used the update rule describe in Ea. J25> to find the min- 
imum of the free energy of the Hamiltonian H described in 
Eq.O- 

A. The ability of NMF Approximation 

Figl3 a ) is a phase diagram for the binary model with r = 3. 
In the figure, we only show the 'replica symmetry' solution; a 
more detailed solution has been given by Nishimori & Wong 
0]. The x-axis shows jo/ J, which corresponds to the S/N ra- 
tio in the communication channel, and the y axis shows T j J, 
which corresponds to the decoding temperature. Region 'F', 
which represents the ferromagnetic state (m > and q > 0), 



in the figure shows the retrievable region. 

Fig EJb) is a phase diagram for the analog model using the 
same parameters as for the binary model. Comparing figs|3Ib) 
and[3Ja), we see that the decoding ability of the analog model 
looks better than that of the binary model in the high temper- 
ature region. 

In Fig|3jc), region "A" shows where the analog model is su- 
perior to the binary model, while region 'P>' shows where the 
binary model provides better decoding ability than the ana- 
log model. The binary model is better when the S/N ratio is 
low, while the analog model is more robust with respect to the 
decoding temperature T. 



B. Comparison with Computer Simulation 

The theoretical results described in sec lHI Al were realized 
in an equilibrium state, so we confirmed the validity of our 
theory through the computer simulations. In the simulations, 
we assumed a message length as N — 100. 

In the binary model simulation, we used asynchronous 
Glauber dynamics. First, we randomly selected one site I, 
and calculated the transition probability as determined by the 
heat-bath method: 

W(a* - = l + expW{<T t ) _ H{HtT t ))) y 

(35) 

where the operator flipped a\ to — a\. The difference of 
the Hamiltonian Uia 1 ) — H(J-i(cr t )) can be denoted as 

H{(T t )-H{Ti(T t ) = 2aiFi{cT t ) (36) 
Fi(<t) = * Ju 2 - lr vt 2 ■ ■ ■ < + h n (37) 

82, •••ir 



7 



Stochastic Binary, yj=0.i 



Stochastic Binary, yj- 1.00 



1.2 



0.4 
0.2 








B 


nary, Simulation ■ — ■ — ■ 
Binary, Theory 


I T.J f 


.1.., 










LI : 












































II 


WW 


1 1 1 I 



0.5 1 1.5 £ 

(a) Binary, j /J = 0.80, N = 100 
Deterministic Analog, V./=0.80 



2.5 







Analog, Simulation ■ — ■ — ■ 
Analog, Theory 


I'll] 




~i~Hf 

























































0.5 1 1.5 2 

(b) Analog, jo /J = 0.80, AT = 100 



2.5 



1.2 
1 

0.8 
i 0.6 
0.4 
0.2 








B 


nary, Simulation ■ — ■ — ■ 
Binary, Theory 


I-i--*--*-J 



























































III! 


1 1 E ] 



0.5 



1.5 



2.5 



p-'/J 



(c) Binary, j /J = 1.0, N = 100 
Deterministic Analog, j(/J=1.00 



0.8 
S 0.6 
0.4 
0.2 








Analog, Simulation ■ — ■ — ■ 
Analog, Theory 


























































— ; t 





0.5 1 1.5 2 

(d) Analog, jo/ J = 0.80, N = 100 



2.5 



FIG. 4: Comparison with Computer simulation. The error bars in each figure represent the computer simulation results. The horizontal axis 
shows the decoding temperature and the vertical axis shows the absolute value of the overlap M . Each error bar range represents the first and 
third quartile deviations, and the middle point indicates the median. The dashed line in each figure shows the replica analysis result. 



This is equivalent to determining the Zth site probability by 



_ _ 1 tanh(/?JK<r«)) 
' 2 2 



(38) 



In this simulation, the initial state er 1 was set to the true mes- 
sage £, so the simulation result indicated the stability of the 
true message in the model. 

In the analog model simulation we used the synchronous 
update rule describe in Eq..j25>. Thus all units were updated 
simultaneously. In the analog simulation, we also set the true 
message £ as the initial state s 1 . 

Fig s IUa) and (c) show the binary model simulation results 
along with the theoretical analysis results at jo/ J — 0.8 and 
jo/ J — 1-0, respectively. (The horizontal axis shows T/J, 
and the vertical axis shows the absolute value of the overlap 
| M Q | .) The computer simulation results are represented by er- 
ror bars showing the quartile deviation, and the dashed line 
shows the theoretical analysis result. The phase transition oc- 
curred at about T/J =1.0 and T/J = 1.5 in figsEta) and (c), 
respectively, and the results of the computer simulation agree 
with the theoretical analysis results. 

FigsEJb) and (d) shows the analog model simulation re- 
sults along with the theoretical analysis results at jo/ J = 0.8 



and jo/ J = 1.0, respectively. The computer simulation re- 
sults are again represented by error-bars, and agree with the 
replica analysis results. In figslHb) and (d), the theoreti- 
cal results show that the phase transition occurred at about 
T / J = 1.5 and T/J = 1.9, respectively. The computer sim- 
ulation agreed with the analysis at such high temperatures. 
In the analog model, the critical temperatures, which cause 
the phase transition, became higher than those of the binary 
model. This indicates that the deterministic analog decoding 
model is more robust than the binary decoding model in terms 
of decoding ability when the receiver overestimates the de- 
coding temperature. 

However, at low temperatures in the analog model the sim- 
ulation results did not agree with theoretical result (FigOb)). 
The simulation initial state is s 1 = and there exist meta- 
stable states around the £, so the dynamics of s* is captured 
by this state and the absolute overlap \M a \ stays close to 1.0. 

An important feature of the analog model simulation results 
shown in figs|4lb) and (d), is that the absolute value of the 
overlap \M a \ was exactly at high temperature. In the analog 
model, the all units value Sj exactly converged to for any 
messages {£}, so we can guess whether the retrieval is failed 
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FIG. 5: Convergence of the macroscopic quantity m(t): the solid 
line represents the analog model and the dashed line represents the 
binary model. i ra j„ = 1 

or not at high temperature exceeding the retrievable limit. In 
practical, the ability to determine whether decoding will be 
finished in failure or success is a desirable feature. 



C. Convergence Speed 

The main advantage of the NMF approximation is the low 
calculation cost of convergence. Here, we discuss the conver- 
gence time that should be set in the computer simulation for 
each method. To calculate the thermal average in the com- 
puter simulation, we implemented 

T = t - t min + 1, (40) 

as the thermal average, where the superscript t means discrete 
time and i m j n means the beginning of thermal average calcu- 
lation. First, we observed the macroscopic quantities to*; 

m t = [|E (binary model) 

\ jt X} S l (analog model: NMF approximation) 

(41) 

In this simulation, we set the initial value s 1 and er 1 to satisfy 
the overlap M a = 0.8, and i m ; n = 1. Fig|5]shows the dynam- 
ics of m*. The solid line represents to* of the binary stochastic 
model and the dashed line represents that of the analog deter- 
ministic model. The convergence times for the two lines seem 
to be of the same order. 

In the MPM inference, however, we should determine each 
unit's value; that is, the microscopic quantity. Thus, we in- 



vestigated the behavior of the thermal average of unit (a\) for 
the binary model and s* for the analog model. In the ana- 
log model, all units were updated simultaneously because we 
adopted synchronous updating. In contrast, we adopt asyn- 
chronous updating in the binary model. To compare the con- 
vergence time between analog unit and binary unit, we re- 
garded N updates as one Monte Carlo step (1 MCS) for binary 
model, where N(= 100) means the number of units in the 
simulation. Thus, one MCS update corresponds to one syn- 
chronous update in the analog model. Fig. |6ja) shows a typ- 
ical result regarding the convergence speed for the S/N ratio 
jo/J= 1.0 and temperature Tj J = 1.0; we set the beginning 
of the thermal average calculation as i m ; n = 100 MCS. The 
horizontal axis shows the calculation time measured by MCS, 
and the vertical axis shows the value of a\ and s\. The solid 
line shows typical dynamics of <7q, and the dashed line shows 
typical dynamics of Sq. In macroscopic points of view, each 
model achieved the same overlap \M \, but in microscopic 
perspective, the deterministic analog model converged about 
over 1000 times as quickly as the stochastic binary model. 

Fig. |6jb) shows the dynamics at a higher temperature 
Tj J = 2.2. (a[ ) did not converge to 0; however s{ seemed to 
be converging to exactly in the early time steps. Thus, when 
using the analog model, we can easily predict within the early 
time steps whether the decoding will be finished in failure or 
not. 



IV. CONCLUSION 

In this research, we have investigated an error-correcting 
code that uses the MPM inference. Since the MPM inference 
requires many trials to calculate the thermal average for each 
unit, we tried to replace this operation with a form of deter- 
ministic analog dynamics called naive mean field approxima- 
tion. We analyzed the decoding ability with the deterministic 
analog model through the replica method, and quantitatively 
compared it with that of the stochastic binary model suggested 
by Nishimori & Wong|6] . We found that the decoding abil- 
ity of the deterministic analog model is superior to that of the 
stochastic binary mode 1 at higher temperature area. To con- 
firm this result, we carried out a computer simulation for each 
model and obtained the results that agreed with our analysis 
results. 



Acknowledgement 

This work was partially supported by research grant 
No. 15700192 of the Grant-in-Aid for Young Scientists (B) in 
Japan. 



[1] D. C. Mattis. Solvable spin systems with random interactions. [2] C.E. Shannon. A Mathematical Theory of Communication. Bell 
Physics Letter A, 56A(5), April 1976. 



9 



1 

0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 



0.2 



Analog unit: s (t) 

Binary unit: (a )(t) 






M 


- 










yy 









100 



1000 

( 

(a) 



10000 



1 

0.8 
0.6 
0.4 
0.2 


-0.2 



-0.4 





Analog unit: s (0 

Binary unit: <ct >(0 






it 




if 

£\l .J A 












i 





100 



1000 



10000 



(b) 



FIG. 6: Typical dynamics of (a[} and s\. The solid lines show the dynamics of s\ which converged within t mm = 100 iterations. The dashed 
lines show the dynamics of (a[). In (a), the S/N ratio and temperature parameters were set to jo/ J = 1.0 and T/J = 1.0, respectively. The 
convergence of (a{) was slower than that of s{. In (b), the temperature was set to T/J = 1.80; i.e., outside of the retrieval region. In this case 
the convergence of (a\) was also slow. 



System Technical Journal, 27:379-^123, 623-656, 1948. 
[3] Y. Kabashima and D. Saad. Belief Propagation vs. TAP for 
decoding corrupted messages. EuroPhys. letter, 44(5):668-674, 
1999. 

[4] P. Rujan. Finite Temperature Error-Correcting Codes. Physical 
Review Letters, 70(1 9): 2968-2971, 1993. 

[5] N. Sourlas. Spin Glasses, Error-Correcting Codes and Finite- 
Temperature Decoding. Europhysics Letters, 25(3): 159-164, 
1994. 

[6] H. Nishimori and K. Y. M. Wong. Statistical mechanics of im- 
age restoration and error-correcting codes. Physical Review E., 
60(1): 132-144, 1999. 



[7] A. J. Bray, H. Sompolinsky, and C. Yu. On the 'Naive' Mean- 
Field Equations for Spin Glasses. J.Phys.C: Soliid State Phys., 
19:6389-6406, 1986. 

[8] J. J. Hopfield and D. W. Tank. Computing with neural circuits: 
A model. Science, 233:625-633, 1986. 

[9] H. Shouno, K. Wada, and M. Okada. Naive mean field ap- 
proximation for image restoration. Journal of Physical Society 
Japan, 71(10), Oct. 2002. 
[10] M. Vecchi. S. Kirkpatrick, C. Gelatt. Optimization by simulated 
annealing. Science, 220(4589):671-680, 1983. 



